You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-11 - 07:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Thu Dec 11, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
505164213159,67sleep30-21swapper/319:05:083
97058921120,3sleep0221rcuc/000:30:170
50978428153,20sleep00-21swapper/019:05:460
50995628064,9sleep20-21swapper/219:07:562
51009627955,19sleep10-21swapper/119:09:431
510258992522,1cyclictest0-21swapper/323:40:003
510258992521,2cyclictest0-21swapper/323:24:593
51025899242,16cyclictest764649-21taskset22:25:133
51025899242,14cyclictest718584-21taskset21:55:183
510258992421,1cyclictest0-21swapper/322:48:543
510258992421,1cyclictest0-21swapper/321:50:193
510258992419,2cyclictest0-21swapper/323:55:233
51025899232,13cyclictest973147-21taskset00:30:323
51025899229,7cyclictest821649-21taskset22:59:363
51025899221,17cyclictest0-21swapper/322:11:183
510258992210,9cyclictest755913-21turbostat22:24:543
510258992118,1cyclictest0-21swapper/321:34:203
510258992118,1cyclictest0-21swapper/321:34:193
510258992117,2cyclictest0-21swapper/322:05:223
510258992116,2cyclictest0-21swapper/322:15:253
51025899211,16cyclictest564640-21taskset19:55:243
9666142201,2sleep10-21swapper/100:27:461
51025899202,13cyclictest958458-21taskset00:21:263
51025899202,12cyclictest976182-21taskset00:35:053
51025899202,12cyclictest806052-21taskset22:50:013
51025899202,12cyclictest706287-21taskset21:50:013
51025899202,11cyclictest567454-21taskset20:00:183
510258992017,1cyclictest0-21swapper/321:25:173
510258992017,1cyclictest0-21swapper/321:00:263
510258992017,1cyclictest0-21swapper/320:50:183
510258992017,1cyclictest0-21swapper/319:40:013
510258992017,1cyclictest0-21swapper/300:05:163
51025899201,15cyclictest515509-21turbostat19:15:003
51025899201,14cyclictest656585-21chrt21:20:143
510258991916,1cyclictest0-21swapper/323:42:063
510258991916,1cyclictest0-21swapper/319:15:163
510258991915,2cyclictest0-21swapper/323:15:283
510258991915,2cyclictest0-21swapper/322:30:013
510258991915,2cyclictest0-21swapper/300:15:003
510258991913,3cyclictest0-21swapper/323:25:173
51025899191,13cyclictest875796-21chrt23:30:223
51025899191,13cyclictest838543-21chrt23:10:063
51025899191,13cyclictest644606-21chrt21:10:363
51025899190,17cyclictest0-21swapper/321:19:183
9604272181,8sleep2960437-21ssh00:25:062
9343442181,10sleep1639289-21diskmemload00:06:051
7910102181,9sleep20-21swapper/222:40:152
51025899182,11cyclictest729522-21taskset22:02:433
510258991815,2cyclictest0-21swapper/320:55:183
510258991814,2cyclictest0-21swapper/300:15:183
510258991814,2cyclictest0-21swapper/300:00:173
51025899181,14cyclictest0-21swapper/323:51:503
51025899181,14cyclictest0-21swapper/323:07:513
51025899181,14cyclictest0-21swapper/322:35:123
51025899181,13cyclictest597342-21chrt20:30:123
51025899181,13cyclictest0-21swapper/323:45:093
51025899181,13cyclictest0-21swapper/320:15:123
51025899181,12cyclictest792665-21chrt22:40:153
51025899181,11cyclictest634321-21taskset21:05:103
51025899180,14cyclictest0-21swapper/323:00:183
51025899180,14cyclictest0-21swapper/321:35:133
9456042172,8sleep1945825-21hddtemp_smartct00:15:151
8284672171,4sleep2639289-21diskmemload23:02:332
8072402171,7sleep2808852-21kthreadcore22:50:172
7816532171,8sleep2781870-21hddtemp_smartct22:35:152
6465422171,3sleep20-21swapper/221:13:372
510258991714,1cyclictest0-21swapper/319:25:173
51025899171,13cyclictest696395-21chrt21:42:423
510258991710,5cyclictest66550irq/131-enp1s0-TxRx-300:25:163
9723582161,8sleep2972405-21ntpq00:30:202
9372122161,10sleep2937231-21sh00:10:122
9266352161,10sleep2926647-21sh00:01:462
8879792161,3sleep20-21swapper/223:40:132
8779482161,9sleep0639289-21diskmemload23:33:060
8540982161,10sleep2854108-21ssh23:19:212
8440492161,4sleep1639289-21diskmemload23:11:261
8300562161,10sleep0830070-21ssh23:04:550
7781782161,2sleep20-21swapper/222:30:452
6963822161,2sleep20-21swapper/221:42:422
6540492161,3sleep1639289-21diskmemload21:17:251
6532582160,1sleep20-21swapper/221:16:182
6396502161,10sleep1639680-21apt-get21:10:111
5406952161,10sleep1542514-21grep19:35:151
5312742161,10sleep2532178-21memory19:25:192
5288762161,4sleep0529532-21cut19:25:170
51025899161,9cyclictest602581-21taskset20:35:003
510258991612,1cyclictest0-21swapper/319:40:023
51025899161,12cyclictest0-21swapper/319:20:233
51025899160,13cyclictest0-21swapper/320:45:183
51025899160,13cyclictest0-21swapper/320:10:203
9818052151,4sleep2981815-21ssh00:36:412
9137982151,9sleep2915415-21kthreadcore23:55:172
9034482151,9sleep0903460-21ssh23:49:080
8777802151,2sleep20-21swapper/223:32:522
8455752151,8sleep2639289-21diskmemload23:13:432
8275632151,9sleep1827574-21ssh23:01:081
7878132151,4sleep0787826-21ssh22:37:540
7449922151,3sleep10-21swapper/122:11:081
7204592151,7sleep1639289-21diskmemload21:56:371
6616012151,9sleep0661612-21ssh21:20:520
6399752151,10sleep0640155-21expr21:10:130
5450712151,9sleep2546544-21cut19:40:182
51025899159,3cyclictest15950irq/132-ahci[0000:00:17.0]20:45:003
51025899151,12cyclictest0-21swapper/320:05:183
510258991511,2cyclictest0-21swapper/319:50:213
51025899151,10cyclictest597083-21sleep20:29:593
51025899150,11cyclictest0-21swapper/320:20:153
9570942141,8sleep1957149-21ntpq00:20:211
9347962140,2sleep00-21swapper/000:06:480
9029792141,2sleep10-21swapper/123:48:251
8391862142,7sleep0839657-21kthreadcore23:10:170
7382732142,8sleep0712757-21kworker/u16:0+nfsiod22:08:310
7126222141,9sleep0712630-21ssh21:52:150
7062262141,6sleep20-21swapper/221:50:012
6564722141,8sleep2656581-21cpuspeed_turbos21:20:142
51025899141,9cyclictest533593-21chrt19:30:083
51025899141,9cyclictest533593-21chrt19:30:083
9739182131,8sleep1973929-21sh00:31:551
9349482135,2sleep20-21swapper/200:07:022
9204682131,5sleep10-21swapper/100:00:011
9116882131,6sleep00-21swapper/023:54:100
8779092131,6sleep1639289-21diskmemload23:33:031
8632062131,3sleep2863475-21df_inode23:25:142
8135972130,7sleep1639289-21diskmemload22:54:421
8041042131,6sleep244-21ksoftirqd/222:47:302
7959282131,6sleep1639289-21diskmemload22:42:431
7781132131,9sleep0778116-21ssh22:30:400
7646172131,5sleep2639289-21diskmemload22:25:132
7388202131,6sleep1639289-21diskmemload22:09:201
7365222131,7sleep2736527-21ssh22:05:522
7167982131,8sleep2718582-21kthreadcore21:55:182
6959252131,8sleep0695939-21ssh21:42:050
6277102136,3sleep0627759-21gcc20:55:230
6134302131,8sleep1613461-21awk20:45:141
51025899130,1cyclictest0-21swapper/319:45:153
9594342120,5sleep20-21swapper/200:23:232
9440492121,7sleep1639289-21diskmemload00:13:311
9424032120,1sleep00-21swapper/000:10:580
9285382120,1sleep00-21swapper/000:04:410
8933532121,4sleep1639289-21diskmemload23:41:161
8796652121,4sleep20-21swapper/223:35:122
8361482121,5sleep20-21swapper/223:06:462
7723252121,3sleep00-21swapper/022:29:440
7546042121,4sleep1639289-21diskmemload22:18:061
7447142121,5sleep00-21swapper/022:10:440
7281732121,5sleep0639289-21diskmemload22:00:450
7142912122,1sleep1171rcu_preempt21:54:451
6982912121,6sleep1698555-21df21:45:111
6695442121,6sleep1669625-21find21:25:271
5812152121,7sleep1581287-21sed20:15:001
9817492111,4sleep1639289-21diskmemload00:36:361
9672702110,1sleep00-21swapper/000:28:540
9529692113,1sleep00-21swapper/000:20:080
9044642110,1sleep20-21swapper/223:50:152
8855012111,4sleep0639289-21diskmemload23:36:530
8694782111,4sleep1639289-21diskmemload23:27:451
8692352111,6sleep0869248-21ssh23:27:230
8625502111,6sleep0862561-21ssh23:24:400
8599852111,4sleep20-21swapper/223:20:452
8467392111,6sleep1639289-21diskmemload23:15:101
8375592111,4sleep0639289-21diskmemload23:08:550
8360202111,4sleep1639289-21diskmemload23:06:321
8194992111,4sleep2639289-21diskmemload22:56:162
8194382111,4sleep1639289-21diskmemload22:56:111
8141762112,4sleep0814195-21apt-get22:55:120
7983552111,4sleep1639289-21diskmemload22:45:161
7950442111,3sleep00-21swapper/022:41:220
7864272111,4sleep1639289-21diskmemload22:35:461
7788552111,4sleep1639289-21diskmemload22:31:481
7551582111,4sleep2639289-21diskmemload22:18:552
7281392111,4sleep1639289-21diskmemload22:00:411
6815752111,3sleep20-21swapper/221:35:102
6815612111,3sleep10-21swapper/121:35:091
6808682111,5sleep0680879-21ssh21:34:280
6808682111,5sleep0680879-21ssh21:34:270
6563542111,4sleep1639289-21diskmemload21:20:051
6368002111,4sleep0638120-21memory21:05:170
6023342111,5sleep1246-21systemd-journal20:35:001
5643302111,6sleep2564378-21ntpq19:55:202
9830282101,4sleep0639289-21diskmemload00:38:520
9515722101,2sleep20-21swapper/200:18:042
9175482101,3sleep1639289-21diskmemload23:55:311
9166122101,3sleep00-21swapper/023:55:180
9085142101,3sleep1639289-21diskmemload23:50:221
8945082101,4sleep0639289-21diskmemload23:43:040
8860412100,4sleep1639289-21diskmemload23:37:441
8549432101,2sleep10-21swapper/123:20:121
8466932101,3sleep00-21swapper/023:15:050
8031722101,3sleep00-21swapper/022:46:070
7697792101,2sleep10-21swapper/122:25:521
7629652101,3sleep20-21swapper/222:23:072
7555692101,3sleep00-21swapper/022:19:340
7287932100,3sleep20-21swapper/222:01:392
7210922101,3sleep0639289-21diskmemload21:57:330
7116082101,3sleep20-21swapper/221:50:442
7061192101,3sleep00-21swapper/021:49:500
6802512101,3sleep20-21swapper/221:33:332
6802512101,3sleep20-21swapper/221:33:322
6802032101,3sleep1639289-21diskmemload21:33:281
6802032101,3sleep1639289-21diskmemload21:33:281
6714632101,2sleep00-21swapper/021:27:500
6650162101,3sleep2639289-21diskmemload21:25:112
6521192101,5sleep0652254-21wc21:15:230
6342992101,4sleep2634318-21apt-get21:05:102
5915052100,1sleep10-21swapper/120:20:531
5748362101,5sleep2574895-21ntp_kernel_err20:05:182
5661142101,4sleep1566185-21irqstats20:00:171
5550822101,5sleep0555132-21/usr/sbin/munin19:50:100
5439282105,2sleep0221rcuc/019:39:590
5323652101,4sleep1532414-21ntpq19:25:231
945186291,2sleep00-21swapper/000:15:030
896046290,2sleep20-21swapper/223:45:022
812023291,2sleep00-21swapper/022:52:050
762250290,3sleep0639289-21diskmemload22:22:060
762039290,3sleep1639289-21diskmemload22:21:461
744959290,3sleep20-21swapper/222:11:052
696185291,4sleep1696198-21ssh21:42:261
685222291,4sleep0685652-21grep21:35:190
633354291,4sleep0633506-21timerwakeupswit21:00:270
630403291,5sleep20-21swapper/221:00:182
628007291,4sleep1628131-21cat20:55:281
611740291,5sleep20-21swapper/220:40:182
592095291,3sleep10-21swapper/120:25:131
590860291,4sleep0590915-21modprobe20:20:190
586513291,4sleep0586585-21munin-run20:20:000
570626292,3sleep0570844-21latency_hist20:05:010
560336290,1sleep10-21swapper/119:55:071
523075291,2sleep10-21swapper/119:20:121
518311291,4sleep0519814-21/usr/sbin/munin19:15:150
510976291,4sleep2511838-21grep19:10:162
638412281,3sleep1638537-21systemctl21:05:221
623238280,1sleep00-21swapper/020:52:060
611921281,2sleep0611968-21ntpq20:40:190
610459281,3sleep1611739-21memory20:40:171
602326281,4sleep0597086-21perf20:34:550
559127281,2sleep10-21swapper/119:50:191
550319281,3sleep1550555-21kthreadcore19:45:171
548656281,3sleep0548769-21ps19:40:240
518019281,4sleep2518232-21hddtemp_smartct19:15:132
633173271,3sleep1633301-21readlink21:00:251
628619270,1sleep20-21swapper/220:59:052
623119270,1sleep20-21swapper/220:50:362
602595270,2sleep10-21swapper/120:35:121
597867271,2sleep20-21swapper/220:30:162
581985271,3sleep10-21swapper/120:15:161
581005270,1sleep00-21swapper/020:12:220
576157270,2sleep20-21swapper/220:10:122
565632270,1sleep20-21swapper/220:00:142
544489270,2sleep10-21swapper/119:40:131
538430270,1sleep10-21swapper/119:32:441
538430270,1sleep10-21swapper/119:32:431
524159271,2sleep20-21swapper/219:20:172
623148260,1sleep10-21swapper/120:50:591
617932260,4sleep00-21swapper/020:46:580
617303261,2sleep20-21swapper/220:45:222
603387260,2sleep20-21swapper/220:35:162
596824260,1sleep20-21swapper/220:27:032
591481260,1sleep20-21swapper/220:20:342
581288260,1sleep20-21swapper/220:15:002
574368261,2sleep10-21swapper/120:05:181
572566261,2sleep00-21swapper/020:05:170
559922260,1sleep20-21swapper/219:53:012
550556261,2sleep20-21swapper/219:45:182
522470260,4sleep10-21swapper/119:16:011
515173260,1sleep10-21swapper/119:11:071
515129260,1sleep00-21swapper/019:10:340
550102250,2sleep00-21swapper/019:45:170
543017250,1sleep20-21swapper/219:35:202
537671250,1sleep00-21swapper/019:30:220
537671250,1sleep00-21swapper/019:30:220
533597250,1sleep20-21swapper/219:30:122
533597250,1sleep20-21swapper/219:30:112
602894240,1sleep00-21swapper/020:35:150
591882240,1sleep00-21swapper/020:25:010
560630240,1sleep00-21swapper/019:55:150
523418240,1sleep00-21swapper/019:20:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional